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ABSTRACT 

A study investigated the effects of specific 
guidelines in the taking and rating of tests of summarizing eUDility. 
The subjects were 63 native-Hebrew-speaJcing students enrolled in 
English-as-a-Second-Language (ESL) courses at the Seminar Hakibbutzim 
Teacher Training College in Tel Aviv (Israel). The subjects were 
given two sets of instructions (guided and minimal) for summarizing 
texts in both English and Hebrew. The guided instructions gave 
specific directions on how to read the texts and on how to write the 
summaries, while the minimal instructions gave traditional minimum 
direction. Two scoring keys were developed from the summaries of nine 
Hebrew-speaking and nine English-speaking experts, respectively. The 
summary guidelines developed in this case did not make a sizable 
contribution to summary development, but an item-by-item analysis 
revealed that the guided instructions were both helpful and 
detrimental. In some cases, the guided instructions helped 
respondents to identify key elements to summarize, but in other 
cases, they may have dissuaded the respondents from including details 
that in fact proved essential in the eyes of the experts upon whom 
the rating key was based. (Author/DJD) 
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ABSTRACT 

T.e .a>n purpce t.e study -as to Jeterj.ne t.e^eHects 
spec.^c ,u>deUnes in a >n,^and t.e^rat.^.^ ^^^^ ^^^^^ ^^^^^ 

sumiarizing ability tests i of their reading 

,nd provide -r.tten /t.^r -^.t.ng abxHty. Other 

comprehension level as «eU a ^ j si.ilarities and 

language of the rater. native-Hebre«-speaking 

The subjects for ^"l- ^Z*^;^,;,', ^ Teacher Training College in Tel 
students from the Se«in,r "'...oficiency English-foreign- 

Aviv. T«enty-six -ere t-o" in ter^edi ate EHU dasses. 

language (EFL) classes an^/^ -ere ^^"^ ,tudy. The t«o 

Four raters a d the students ^^^^^ „ere both native 

:r;rated the Hebre- ----^^ „^ . t^e. su..aries of the EFU 
Hebrew speakers. 0* ' '1.^^ and the other an English 

te.<t-., one -as a native , ,tudv. t-o in Hebre- and 

speaker. F»ve te.ts advanced EFL student being 

three in English '"^^^ /^^^^J % " ostructions -ere developed^ 
„ked to su.«arize t«o • «o sets ho« o ad the 

One version Mas "guided witn SP^-^' version had the 

:ts and ho« to -rite ^''^^ /^"^^ ^e' scor i . keys for the texts were 
typ.cal ".ini^al" '"^^'■"f i°"^-Heb e«- = P"'<'"9 and nine English" 
Zei on .he su««aries o n ne he e« P ^^^^^^^ su««.ari:ed the 

speaking experts '""P"*'"''^-. ' second Hebrew text, and on the 
^l,, Hebrew text, 53 ---'/f,,';;. The students wrote su^.aries 
average, slightly more than a tmra 

for the EFL texts. guidelines developed in 

The study did "o^"^"""*! Yet they did see« to have a 

this case «ade a suable on an ite.-by-ite. basis 

ertain influence. An "^^V-'^^f,;;/: ^ "ed to be both helpful an 
revealed that the guided respondents in finding the 

Jetrimental. In so.e ^;«V 'they prcbably dissuaded 

^(ey ele.ents to su..arize, ' ,° in f-ct proved to be 

fste^uTtrthre-yes-ortU^rpe'ts upon who. the rating key was 

based. 



r,":r.'h"'o".'c'kro'wr.d;rEi.n. s>'f^"|;;;;i,;!^:"co!uct!n;"til^2^a!'"I 
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Summarization tests are complex in nature. In ° 
successfully on such tasks, test takers nee a J^^^^/ . Ta^e^ s 

Irformation -- information that often is not provided. Test takers 

t know the best way to re.d the given text(s) in order to 
prepare the best written summary for the given aud e . he 
oreoaration of a summary outside the framewort of a test is a 

cult enough task; summarizing for test ^-P^^^^^^^^^""^^ , 
difficulty. In summarizing, the reading portion entails 
t p ar formation, distinguishing superordi nate -J-"'"*;^ 
::?;rial, and identifying "dundant and trivia 

C^re^er^t'g": Tlt^^Is^:"' roi^ed'^Slleting trivial and 
edun ant formation, substituting superordi nate material or macro- 
p" s!uo:s, and restating the text so that > "t^;;^,/;^", 
polished (Kintsch i van DiJk 1978, Brown J. Day 1983, Chou Hare 

'"^Tn'I ludl'of high- and 1 ow-pr of i c : ency college-level EFL 
stude s, Johns and Hayes (forthcoming, found 1*''^^^"^^^ 

ru:rhe:re7r lorsrudrntrirrrfo-r;: ;"e ^ 7/1;.^ 

« : t of direct copying of material from the ° ^ 

summaries, as it was "^-;„^-^;-;,;;:/-:d7,"o s^aS^^: ^ad^So^nd 

second language as well. In an earlier siuuy, 

that underprepared natives were likewise more prone to use 

""":*r=r'.:;;;;'s'.r;; n™; .«->..» '"7-*;.'° 

exll cit teacning of 3uch reconceptual i zat i on may not yield such 
no. ive esuts either. Bensoussan and Kreindler (in press), for 

the gist of a text, they still expressed frustration at their 
inability to distinguish macro- from micro-propositions. 

leice summaries are problematic -d yet they eeP |PpeaMng m 
tests of r;.ding and of writing, ^^^^ V - - ap I 
"authpntic" tests, s nee people need to summarize in ine reai 
o! course a real-.orld summary may be quite different from a test 

m R a m.aries are usually prepared for someone who has not 

reir^he text and -P^^^nts to know what > - --^Je.t ^^^^ 

:reTe:rir:brurrnrwrn;s'to"s::"t:what extent t pondents 

•"";;;re%r;hr;rob!:ror;;um..ry t.sk . ^ -id measure of 

,e a further problet of rf?liaoiiity 
reading conprehension, there is a turtner H^uuic. 
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. 1 fha uavc in which respondents at ditterent 

its purpose to investigate the ways in wnicn re y ^^^Hmc 
Iroficiency levels carry out su^iarizinq tasks on a reading 

readlnq strategies, including su««.ruing. They '■-P"""^^;^^ 
reaaing sir.i y , course instructors Mho typically rated 

rhrErera.l^:fs;.-:z!;; r^^IM the PUC aUo participated in the 

^^"''%h: ;fn:;:;s sho.ed that t pondents - ^^^^^^/J; ^.f, j ^J",: 

diHi.ulty Identifying topical information, ye they ad ^'^^^ 
d St nguishing superordinate, non-redundant «aterial fros the rest, 
due n Urge Jart to an insufficient grasp of foreign-langua e 
locaiul ry for then -ritten su.«aries, they did "ot need to 
oenera te t;pic information because .11 the texts provided it ey 
d! o KoLver, have a good sense of balance with respect to hoH 
luchtc delete. E.ther they were too vague and general o too 
3e ai d ;*hile there .as some concern for coherence P^J-^'" ' 

he e appeared to b. relatively little attention paid to lucing 
thorough V coherent and polished summaries. In ^^^^^ 

«,th .ain Ideas and connecting schemata for each "^t; 

The «ain purpose of the present study was to determine 

and provide written su.naries « a measure of their reading 
the respondents a. to how to construct their , " , 

^at'thr nece^a^ily exercise this knowledge when -^^^ lasting 

i Uation. In addition, the rater, .ay also lack specific nough 
guidelines in order to rate the ^"""^'■lesrel lably. OtKer issues 
2n er con.ideration in this study included ^'-'^'^'V ' " oreign- 
diff.r.nc« in native vs. foreign-language -...ri s, « 
language proficiency level of the - ; J 'rhe"nat;ve 

between the language in which the su««ary was writ.B 
language of the rater. 

The study asked the following questions: . „„ ^ 

g 1 In wL way, do guid.d instructions affect performance on a 



'"'what extent does the language of the -"^"^^^^^ ^/J^,, 
roie en the sum.ary is written in ^.e respondent s na ive uaQ^ 
3 What IS the relationship between indices of EFL P"*'""'^ 
e EFL course level/course gr ade/uni versi ty entrance exam score) 
d ; llity to su««ari:e in a foreign and a native language^ 
* 4 HOW does the native language of the r.ter """^ 
rating perfor.ance on the summaries, and how consistent are the _ 
ratings across raters? 

Design of the Study 

University ana ^.ne hiahlv proficient in Hebrew, as 

I^e'raiV^eceirer :;ivr e Tf "s^.o^in^^ a Hebr.w day .chool in 
thl S S. wir^ar^Id to'an Israeli, and J, ad lived in Israel .or five 
years. 



Instruments 

The Tests: 



Five texts were selected Tor the .tudy, two in 
Hebren and three in/English. The two Hebre. texts wer. intended to 



Hebrew and tnree in/cMy»*=» - - ^„nfon*- 

reflect two ieveU ot difficulty - both i. er.s ^^ ^-^[l^^ 
romolexity of language, and suffl«Ar i zabi I ity (i.e., how easy they 
coapiexity OT 1 y ^ » Hphrew text (800 words) was 

aopeared to suaiiar ize) . The first Hebrew text low ^ 
dpH««»' ,- , 1. J - irnm A Plav Thino to an Art Pom. it 
entitled, "The FiU Media — from a f'lay «ninq . . *.u.,- 

::Lr:r.s:.i:r;;:^"re:ry:"rhe"s::rd';^ar^ 

e - he Proble.aticity of Modern Israeli P^ose - ro. 
Detachment to Continuity." This text -as "^'^"^^ / , 
problematic style, and a = 't, 

certain degree of background knowledge about the topic. 

dffi::;ry^^i;e-ei:i::;%!roro':;r::>:\:tiurd- 

Zl^ollT. ted a series of P — - -ond 

"h in'language a^d in its "-^P^-j d \ be the 

article (850 words), entitled "Specialization, . , 

Tost d^Hicult articl., but actually turned out to have a relatively 

-^--l^,!°:.^r""sl:r::!:^; two sets of instructions were 
developed.^lersion was "->;ed«2ith specific --uc ,ons on ow 

:n-i:^:;:.-"r;u"ded;:;,tructi.^^^ to. 

had tne ^yP^"» ^^^^^ to identify the nost 

;:;o:raSr;";t%'--^"o"th'at contained the .ey sentences in each 
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paragraph or those that Hould «ake the suasarv interestino to read. 
T.,e respondents Here then instructed to nr.te the su««ary such t 
the content «ould be reduced to only ths essential points and that 
1.8S Important details or those detracting *ro« the .ain points Hould 
bt eli.inated. Thev nera also requested to nrite brieHy I'l'^ 
120 Hords per sugary. » Their te.t nas to comprise one paragraph nith 
all the Ideas linked bv connecting nords. They nere requested to 
Hrite th« suM*riM in their onn Hords. and in the case of HebreH 
su.A.ri.s oi EFL te.ts, not to translate nord for Hord. They nere 
also asked to nrite a draft first and then to copy it over legibly 

(see Appendi-: A). , v u vu. 

The .ore traditional, non-guided instructions sisply to.d the 
respondents to read each text so as to be able to nrite a su««ary of 
,t. They Here asked to be brief -- BO-120-^ord su.maries, to Hritf 
their su.nary first in draft foro and then to copy it over legibly. 

r ,cfr..rfinn nf Rat ino Keys ; The tHO Hebren texts were read 
and su.«arized by nine Hebrew speakers, all experts xn the areas of 
reading and Hritinq. Three Here university lecturers, specaliiing in 
discourse. The rest -ere university students of languaoe arts and 
teachers in their onn right. The three EFL texts nere read and 
suMarued in English by nine native English-speakers, «ost of Hhom 
Here university instructors of EFL. The summaries of these experts 
were anal/zed and a key nas constructed to include only those mair, 
Ideas and linking ideas that a majority (i.e., five or «ore) of the 
experts had included in their sun«.aries (afte- Sang 1989). In thi, 
study, no elfort nas nade to distinguish the «acro- from the micro- 
propositions in the scoring key. 

The l.ev for the HebreH texts nas produced by one of the tno 
raters The key for the EFL texts nas produced by the investigator 
and then translated into HebreH bv the Hebren-speaki no rater of the 
EFL texts, -hose English skills nere »lso quite advanced. ihe EFL key 
H-s transl-ted into HebreH. The keys appeared in the ior-, of a ist 
of numbered ideas, each in sentence form. Appendix B shons the level 
of agreeiient for each point selected per passage. 

The su.«,aries of the Hebrew experts reflected an 80/. average 
agreement as to nhich «.ain loeas and connecting ideas should be belong 
;n the su,«ary. The su«.aries of the EFL experts reflected an 3./. 
average aqrpe.ent as to nhich ideas to include. Hence, U should oe 
noted that there was not full agreement even among experts as to -hich 
Ideas were essential to the construction of a meaningful summary. 

nr. tenon Measures ; Measures also included t«o pieces of 
oackground information on all the respondents - the score that they 
received on the Psychometric (college-entrance) Exam and the mid-year 
grade that they received in their EFL course. 

Data Collection Procedures 

The students Here requested to write the summaries in tno 
•eetinrs of their EFL course - the first one for the Hebrew texts, 
the second one for the English texts. The sessions took pUce in 
December, 1988, and were one neek apart. Each lasted approximately 1* 
hour*. The task Hith guided instructions Has administered to every 
other student according to hon they were seated in i given clabsroom. 
The other students received the unguided version. A. students 



"The stipulation of number of nords is based on common practice in the 
United States. In Israel, students are used to being told to nrite a 
paragraph or a page (a Itsf precise measurt). 
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received in the second sitting the same type of instructions as they 
had received in the first, as this time their name nas written on the 
instruction sheet in advance. The tasks nere not p-esented as 
obligatory and nere not to count tonards the students' grade in the 
course. Fifteen percent of the students did not summarize the second 
HebreH text because they found it too difficult, and about 65X did not 
summarize the EFL texts for a similar rsason. 

All 63 respondents summarized the first HebreH text, receiving 
guided instructions and 31 not. Fifty-three respondents summarized 
the second Hebren text, 27 receiving instructions and 26 not. On the 
average, only a little more than a third of the students nrote 
summaries for the EFL texts. A breakdonn of the numbers of 
respondents for each text according to version (guided vs. unguided) 
and respondents- level of proficiency (intermediate vs. advanced) 

appears in Table 1. ^ ^ j j j 

Altnough the instructions for both the guided and ungu.ded 
versions specified that the students nere to nrite a draft summary and 
then copy it over neatly, only seven of the 67 students did this for 
the Hebrew summaries, four of the intermediate studsnts and three of 
the advanced ones. As for the summaries of the EFL texts, on y one 
advanced student Hrote a draft, as that student had done for the 
HebreH texts as weil . Thus, this element in the instructions was not 
folloHed in either of the tHO versions. 

Data finalV Bi'T Procedures 

The raters received brief training bv the investigator in rating 
the summaries and thei did several of the ratings in the presence of 
the investigator in order to resolve problems of immeJiate concern. 
The rat»rs Here also asked to make note of problems to be resolved in 
consultation -ith the investigator after completing the ratings. Fach 
^ain Idea and Unkidq idea received one point in the rating Process. 
No effort nas mad« to have the pairs of raters of the Heoren and ErL 
texts ^conference" together Hhile learning hoH to do the "-atings. 
HoHever, ratings of the tno pairs of raters for the Heb'e« and EFL 
texts were correlated to determine interrater reliability and 
discrepancies nere identified on a point by point basis in order to 
determine the types of ideas for which raters had difficulty reaching 

consensus. -nrc on ^„ 

Crosstabulations and ANOVA were run using the SPSS-PC program to 
check for differences in success at summarizing between the 
-aspondents given the guided and unguided versions, and between those 
at the intermediate and advanced levels. The Correlations prrfqram was 
run to determine interrater reliability and to assess the relationship 
between summarizing and other variables - i.e., performance on the 
psychometric -«am (an entrance examination for Israeli universities) 
and the respondent?' f i rst- semester grade in their EFL course. 

Findings 

I . The Effect of Suided Instmrtions o n Summanzino 

T^Tha Rn mmanes of Hebrew Texts ; Overall, guided 
instructions did not seem to enhance summarizing. In fact, on the 
first Hebrew text, the one on films, the second rater rated the 
unguided group significantly better than the group with guided 
instructions. Yet when behavior was examined at the idea-by-idea 
level, some interesting differences emerged suggesting that the 
provision of guided instructions has a differential effect on 
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sumiarizing, perhaps helping in soffe instances and interfering in 
others. In exaeining the eain ideas Hithin that File text, itee #5 
was one ior which the unguided group were rated significantly better 
by both raters (see Table 2). This idea involved giving details as to 
why file is an art form (e.g., it deals with shape, color, ecvenent, 
words, and ausic). It is likely that those getting the instructions 
warning thee not to include unnecessary details were reluctant to 
provide this level o* detail for fear that it would be rated as 
extraneous. 

On the second Hebrew text there were two significant differences 
by version, one tavoring th& guided and one favoring the unguided 
instructions. In the first instance, the guided group which was told 
to identify all the eain ideas in the key sentences included idea 16 
■ore than did the unguided group. This idea consisted of the second 
of two txaaples of "detacheent and then reintegration in Isra^M 
prose." In other words, the guided group was aore s^^nsitive to 
incluQing both of the examples. In the second instance, the unguided 
group was eore likely to include in their sueiDary a linking statement 
to the effect that modern Israeli prose is characterized by continuity 
in the nidst of apparent se^aratisffi (see Table 2). In this case, it 
was the guided group that was reninded of the importance of linking up 
ideas and yet the unguid?d group practiced it more successfully in 
this case. 

b. The Summaries of EFL Te?:ts ! In the EFL text ^UFimcries, 
tnere were two instances of significant difference by version, 
according to tne fi st rater. The guided group were niore litalv to 
include the first idea in the summary of the Hodern Constitutions text 
— namelVi the ide^ which provides an historical perspective for the 
passage. Then the ungui(i?d group was r*tore likely to include the 
detail that countries differ as to the number of special checks and 
balances stipulated by their constitution (see Table 2). Hence, the 
group given the fuller instructions seemed more sensitive to include 
the introductory idea while they once again appeared to be reluctant 
to -include detail that might not be deemed central to the summary. 

2. The Lanquaoe of the Text and Nat i ve-Lanouaqe Suamaries 
in thi^ study, performance in summarizing native-language texts 
appeared superior to that of performance on EFL text sumaaries. Just 
considering those subjects who wrote summaries in both languages, the 
overall mean for the summitries of the Hebrew texts was significantly 
higher than the overall mean for the summaries of the EFL texts 
averaged across raters (see Table 3).' Thus, the writing of native- 
language summaries for the EFL texts could be construed to be £ more 
difficult task than writing summaries for the Hebrew n Ui ve-H.iguaoe 
texts. For one thing, writing the summary in the native language 
eliminated the possibility of lifting material directly fron the 
passage without fully understanding it, thus making the summarizing 
task more difficult. 

While the two raters of the EFL text summaries had similar 
average total ratings for the three texts separately and overall, the 
two raters of the Hebrew tex'ts differed, with the first rating more 
leniently (see Table 3). In addressing question #4, below, we will 



Q le analysis still remains to be done, namely, to submit the results 
main idea in both languages to a Rasch analysis to determine 
ither these itees fall on the same unidimensional scale. 
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louk .t the ..in Idea, that raters characteristically agreed upon and 
disaq'.ed upon in their ratings. 

3. TK, Poi.t.nn.hio ^ l"d ,ces of FFI Prof.ciencY an. 

'-^^**^-?=^hat respondents' EFL co.rse level «as "ot reUt^d 

to their Heb carizing ability on an -era asi 

int»r.ediate and the advanced groups scored * ' ^ °" l.^asi s 
according to both raters (see Table 4). 0" '^'^'-''^ ^" . 

,.dge. the intermediate -^P-;-;:,,^:„-^ ^ " , e^ ^he advanced 
significantly more often, and the second rater 

;:r!i!r:rin"a^7or;;:r, s-election. .ocus, and editing, 

•'•'":.ti'respect to the EFL su«.aries, ^^-^^ ' /^//^/^E^Texts 
inter.ediate-level students «ho -rote su».aries ^"^^^^"^i^ f^'^, 
I ..oH th= 'pn advanced students Mho responded (see labie ti. 

Pasier one (on Foolish Opinions). This can help eKpla-n the 
diHerence in ^avor o^ tne '"^^^J'^J'f ^ rhirS'idea on the Constitution 

the constitution .as put a ove ot - ^ t,T,Lt m their 

s:rr:e"';iirrnro r'udrntl-ln'^he advanced EFU course 00, 

dTnl^M;:^ It «ould stand to "-^J^-^.^^^r.rrar a.es 
students -ould '"^lude th.s t " U .ay be that 

coneone an advanced student is caliea iniu h 

advanced students kno« »orr grammar and vocabulary, bu J" ""J 
necesslrily have better cognitive po-ers of discernment and choice 

when It co«es to summarizing a "^P^^^^^;;- course and their EFL 

ThP rcsDondents semester grade m tneir cri. ^ 
.rade on i^e^uni ver si t y entrance e.a. (the ^^Vcho.etr «ere al.o 

the cognitive abilities caii-L Tur ruurses or oo 

the sa.e ones needed to obtain high grades in the EFL courses 
well on the EFL subtest oi the entrance exain. 

4. TK , ,((.rt of fi'^'r'. Native Lannuag p nn Summary RatwL<ls_ani 

Ri^tpr interreliability ^^^^ ^wo 

In answer to the second question above, it "as 
f Till EFL texts had similar total ratings (r = .85, p<.001), 

"f- thH^e-rot rons!:trcy%ror^ 

English speaker V.t -hen «e ° '^^^^ ''^^ / , there are marked 
idea to Idea -ithin thp three EFL ^^x^^' ..^g,, agreed on 
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stcond, or the ourth (an extension of the second suggestion). 

On the ficond text, the raters agreed on the opening statenent 
which gives an historical perspective and on three oi the basic points 
(i.t., a ju*tj f i catior, for constitutions being above law, protection 
of special coaaunities, constitutions of countries differ), but 
disagreed r, ths topic sentence linking the passage together 
cohesively (in teras of the constitution as a fresh start) and on an 
ite« dealing with details of what rights a constitution includes. On 
tht third lfxt,-the rattrs agreed on all but the opening idea (i.e., 
that by the end of the nineteenth contury the intellectual generalibts 
had given way to specialists). 

Thus, the raters were inconsistent in their ratinns ot several of 
the tore global, linking ideas and of ideas involving details :n the 
EFl texts. These discrepancies would not appear to be the result cf 
native language difference?. Instead, it would seee that these are 
perhaps the ideas that lend theiiselves to the nost controversy in 
rating, even when a precise key is provided. 

For purposes of comparison, let us look at the intercorrelati ons 
of the ratings of the two Hebrew native speakers on the two texts that 
they rated. On the File text, there was significant agreement on five 
of the points, four involving basic description and one involving 
basic exeeplif ication (i.e., what eakes file an art form) (Table 6). 
There was inconsistency on three points, two involving contrast (both 
involving the contrast between then and now and one nvolving 
detailed exeapUf ication (i.e., regarding modern technology in eoving 
■aking). 

On the Modern Israeli Prose text, there was considerable 
agreement involving the opening contrastive idea U.e., separatism and 
integration), two examples of this, and a linking idea 'i.e., dealing 
with the continuity in Israeli prose) (Table 6). There was some 
agreement concerning a descriptive point regarding the origins of 
separatism, and less agree^jent regarding two other descriptive points 

a point about continuity and a point introducing the two examples 
of separatism and integration. 

This study did not attempt to investigate why raters disagreed on 
their assessments of respondents' answers as conducted in a previous 
study (Cohen 1988), but the findings would indicate that certain ideai 
on a rating kev are problematic for scoring and others are not, and 
that the language of the rater may not be as important here as other 
factors. 

Discussion and Conclusions 

This study had as its main focus to investigate the contribution 
that instructions make to a test of summarizing ability. Previous 
research on test-taking strategies would suggest that instructions on 
tests may not even be read if they are heeded to at all (e.g., Cohen 
1984). The intention in this study was to build i.ito the test a set 
of instructions that would serve as a genuine guide for the 
respondent. It h^d been the experience of the investigator and of 
colleagues that test instru t*ons were i^^en notoriously vague and 
that they presupposed an understanding o-. -low to do the task 
especially in the case of an activity as complex as summarizing. 

The current study did not demonstrate that the guidelines 
developed in this case made a sizable contribution. Yet they did seem 
to have a certain influence. An analysis of summar .s on an item-by- 
item basis revealed that tiie guided ins*:ructions ap ^ared to be both 
helpful and detrimental. In some cases, they assisted respondents in 
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finding the key elements to summarize, and in other cases they 
probably disiuaded the respondent from including details that in fact 
proved to be essential in the eyes of the experts upc.i whom the rating 
key was based. 

In addition, performance in summarizing nat i v ?-l anguage texts was 
superior to that of performance on EFL text summaries. It was also 
found that respondents' EFL course level was not -elated to their 
Hebrew summarinng ability on an overall basis. Bolh the intermediate 
and the advanced groups scored similarly on the average, according to 
both raters, finally, with respect to mterrater consistency, it was 
found that the raters differed in their ratings of se'eral of the more 
global, linking ideas and of ideas involving details in the EFL texts. 
These discrepancies would not appear to be the result of native 
lenguage differences. Instead, it would seem that these are perhaps 
the ideas that lend themselves to the most controversy in rating, even 
when a precise kuy is provided. 

Limitations and Suggestions for Further Research 

Although the intention was to indicate for each summary who the 
text was being summarized for, th^s was inadvertently left out of the 
instructions for both groups. If we take the EFL text on 
Specialization, for example, it was intended that the respondents 
would be told, say> that they work for a company that nakes 
documentary movies and that their boss has asked them to read an 
article on the dangers of over spec i al i zat i on and to summarize it for 
him in order that they may get some ideas for the preparation of the 
script for the documentary. Perhaps the addition of this modicum of 
functionality would have prompted more ^he students to do the task 
in a context where participation was not obligatory. 

It may also have been beneficial to the research ^o have to 
sum.naries coant for course credit. Then perhaps the students would 
have taken the EFL text summaries, and even the secon'^ Hebrew text 
summary, more seriously. The fact is that the respondents -nay not 
have behaved on these tasks as the.y »<ould on a genuine test. 

The first rater of the Hebrew texts had an observation regarding 
the scoring key. She felt that by using a key based on the judgment 
of "experts," it was skewing the assessment away from the level ,of the 
students who were being assessed. She felt that the experts wrote a 
different type of summary from the seminar students more of a 
logical sequence in the ideas, a tighter use of words, and a greater 
quality of writing. Besides for these observations, she noted thdt 
there were points that the experts did not agree upon and so they were 
left out of the key. Yet students felt these points to be important 
enough to include. compromise would be to build a rating key based 
both on the suggestions of the respondents as to key ideas and the 
insights of the examiners, as was done by Bensoussan and Kreindler (in 
press) . 

In the work by Bensoussan and Kreindler* as soon as the 
respondents finished summarizing the given texts, they were presented 
with the two researchers' set of main for the summary arrived at 

through conferencing, and were asked as a ;roup to react to that set. 
If they disagreed with any points, they had to convince the teacher 
researchers that a change was in order in the scoring key. Apparently 
the students became more proficient at this task as they performed it 
various times. While at the beginning of the course the teachers 
dictated the correct summarief, by the end of the semester, the 
students had learned how to suggest changes (Bensoussan, Personal 
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ConBun i cati on) . ««iio#.f»aH m this study as had 

Fin.Uy. no test -taking ' % Cohen 1984, 

bttn collected in .tudie, reported on ^ ""J^" j'^ the students 
1966). Henc. there -as no '""-^"'^l 7o ^e eUborated 
taking the guided version P"".*"^ " ^ „ bu to read those 

instructions that they Judenl received the .nguided 

instructions aloud since every other ^t""^"' regarding the 

instructions. Perhaps a future ^'^J^;//;, /°i'r^oH-up study would 
procMing of the instructions ^J,^ ,n order 

include rater conferencing with rrocc raters. Such 

to alio- for potentially greater "-^^^^^^ ^ /v h;ough 

behavior of the rater. 
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Table 1 

Respondents ior Text Su.^aries 

ilnauided Version 
G.nded Version unguiue 

T 4. 1 Tnt Adv. Total 
Tnf Adv. Total 1"^* " 
EFL Level: int. hov, 

Hebrew Texts; 



Qveral 1 



/V *. 19 1^ 3- 

Fil« as ftrt 

tA 11 27 17 9 26 5^ 

Modern Israeli Prose 16 

EFL Texts: 

^ 1 ^ - 1 ' 

Foolish Opinions 12 " - 

R - 11 4 7 11 - 

Modern Constitution 8 



3peciali::ation 



4 7 1^' 



15 
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Table 2 

SuAnaries Hebrew and EFL Texts by Version 
(with/without Guided Instructions) 



Rater 2 - Total PerK Rater I - Idea 5 Rater 2 - Idea 5 



FiU 

as 

Art 

Text 

in 

Hebrew 



Poi nts 

0 
2 
3 
4 
5 
t 
1 



Ver SI on 
Guided Unguided 



4 

13 
6 
4 



IB 
7 
I 



(Details) 
Version 
Pts. Buided Ung. Pts. 

0 26 15 0 

1 6 16 1 



(Details) 
Ver SI on 
Guided ling. 
28 16 
4 13 



N = 32 31 
d.-f. 5 
Chi-Square=15. 19»« 



d.-f. I 
Chi-Square=7. 4B»» 



d.-f. I 
Chi-Square=6.92»* 





Rater 1 - Idea 6 


Rater T - Idee 7 




(Second Example) 


(Li n^ I ng Idea) 




Versi on 


Ver Si on 




Pts. Guided Ung. 


Pts. Guided Ung. 


Moder n 


0 7 14 


0 15 7* 


Israel i 


1 20 i: 


1 12 19 


Prose 






Text 


N = 27 26 




1 n 


d.^. 1 


d.-f. 1 


Hebrew 


Chi-Squflre=4. 32* 


Chi-5quare=4.47* 


. #* Pv 


.01 






.05 






Rater i - Idea 1 


Rater 1 - Idea o 




(Perspect I ve) 


(Detail ) 




Vers« on 


Version 




Pts. Guided Ung. 


Pts. Guided Ung. 


Modern 


0 0 5 


0 6 10 


Const I - 


1 11 6 


1 5 1 


tut I on 






Text 


N = 11 11 




in 


d.-f, 1 


d.K I 


EFL 


Chi-5quare»6. 47«* 


Chi-Square=3.67* 



♦ p\.05 



ERIC 
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Table 3 

Ratings by Language of Text and by Rater 



N Hebrew re::ts S.D. 
Ave. X Score 

Raters ^ Texts 

Conbined 27 51.73 14.06 

Overall Hebrew Texts 

Rater 1 63 56.93 14.46 

Rater 2 63 46.34 14.63 

Fill Text 

Rater 1 63 66. 47 16. 1 6 

Rater 2 63 50.20 14.64 

Prose Tex't 

Rater 1 53 54.72 19.36 

Rater 2 53 49.86 20.32 

Cueral] EFL Te;ti 
Rater 1 27 
Reter I 27 

Dpinions te;^t 
Rater 1 16 
Rater 2 16 

Consti tut I on Tex t 
Rater 1 12 
Rater 2 22 

Specialization Tex t 
Rater 1 7 
Rater 2 7 



EFL Texts S.D. t d.-f. 
Ave. X Score value 

35.06 ^ 14.86 4.78**# 26 



7.8U** 62 



7.27*** 62 



2.90** 52 



35.39 15.57 .38 26 

34.77 15.30 



66.75 18.23 -.5«J _15 

71.43 20.87 



46.75 17.73 .20 21 

46.10 16.46 



6i'.71 28.35 1.92 

46.43 9.45 
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Table 4 

Overall SuMarizing Ability by EFL Ccurse Level 



EFL Suiiaaries 
Intermediate Advanced 



Mean 
7.59 



S.D. 

2.50 



7.53»»» 2.70 



Hebrew Suflinaries 
Iftteraediate Advanced 

Mean S.D. Mean S.D. 

Rater 1 B.B9 2.14 8.04 2.14 

ANOVA F Ratio 2.42 

Rater 2 7.24 2.35 6.95 2.22 

ANOVA F Ratib 1.55 

«««p<.001 
««p<.01 



Table 5 

Summarizing Abil.tv by EFL Course Level: 
Crosstabulations DiHerence- at the Idea Level 



Mean S.D. 
4.30 2.00 

12.49** 
4.10 .88 

15.03»»» 



Rater 1 - Idea 2 Kater 2 - Idea . 
(Description) (Description) 
EFL Level EFL Level 

Pts. Inter. Adv. Pts. Inter. Adv. 

ull ; ^5 !l ! ^5 \t 

in 

Hebrew N = 37 26 

d.f. 1 ^ 
Chi-Square=3.98» Ch i -Squar e=5. 24* 

^ p^.05 



Rater 2 - Idea 7 
(Exemplification) 

EFL Level 
Pts. Inter. Adv. 
FiU 0 26 1 2 

Text 1 11 1* 

1 n 

Hebrew N = 37 26 

d.K 1 
Chi-Square=3.71* 

« p^.05 



Rater 2 - Idea 3 
(Justification for 
Constitution over other Laws) 
EFL Level 
F'ts. Inter. Adv. 
Cons- 0 8 10 

stitution 1 4 f> 
Text 

in N = 12 10 

EFL 

Chi-Square=4.07* 
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Table 6 



Correlations of Ratings for the Two Sets of Raters 



EFL Texts 



1 -5 3 4 5 6 7 Overall 

Purpose ist 2nd Ext ^rd 4th 
for Sugges- Sug. of Sug. Sug. 

Op.n- Rater 1 ^Article tion .75.0 - .56^ 

ions with Rater ^ .19 .04 . 
Text 

N=16 



0345 67 
^"^^'perspec- " Justi- De- Descrip- Descr. Link 
tive ficaticn tails tion 

, , ^ .59** - .47 .20 .91** .46 -.05 .o2** 

Consti- 
tut ion 
Text 

N=22 



4 

Opener General lanor. Result 

lanor.nce in Field of Mechanization 
.17 l.vu** .4' -'^ 

Specia-.- 
i rat ion 
Te/t 



^11 fpvfc = 85*** {N=27) 
Correlation ot r;^tings across all texts 



Hebrew Te;;ts 



. . ..4 5 6 Overall 

Idea: i w i 

De- Descr. Con- Descr. Basic Descr. De- Con- 

\ trast Exempli- tailed trast 

-U. .ater 1 .55... .e.o. .Z3 . 59... . B3». . 57.. . . 1 0 . 20 .30 
Text W'' Rater 2 

N=63 



Contrast De- Descr. Descr. Exeinp- E.e.p. Link 

. ..""^ .0. ... 

Tevit 

N = 53 



Correlation of ratings across all texts = .73*** (N«53) 



* pv ,05 
*•» p .01 
**♦ p . , uOl 
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Hf.pendi;: A 

Translation o< the Elaborated Instructions: 

i:.^^ ..n Tp.t and Wr-ti n q a Summary of the Te-/t . 
flfoJe vou are four texU, tno in Hebren ana tno in tngl.sh. 

^^t!rlI^ 'Te\[°co':!'r e><tract the most important points from it, 
; St oints that contain th. .ey sentences (for te given 

Tarlgra^h); or 'hose points that the reader of the summary hiU be 
interested in reading. 

T„ = frurf>onc for Wri M"n ^^'^ Rufinarv iin Hebren) 

r^ R e the n ormation to main points only: avoid the |-lusion of 
:;Jdant L.or.ation. Including this information Hill detract tro. 

Sr'itr'briefiy: the length of the summ.r, is to be 8.". «ords for the 
iircf nassaoG and 120 Hords for the stconc 

- li" the !Larv as a single passage: use connecting nords to 

;'^r:or::;nM:t:t'teranv: .-te the summary m your o.n «ords. 
Write ^ dr. ft first, and then copv it over legibly, 



T-snslctior o- tne Tr„ = iticna; Eri*-' ! t r tiC t : ons : 

;-aHirn fpt T e't a"g writing ^ Cu-^.Tia -'. oiJ± . 
.ariinq -P^ ^ _ ^ tnclish. < c.re 

read e^cn -.rs 5(. as to write r S'-.m^ar v o* i.. 
:n.tr.,rtions f n. far.iino thP ^Hi.,..pr, Mr, HebreH. 

„,;te brief Iv: the length of the sun.mary i= ..o be nords tor 

.irst p-ssace anc \2!^ HOrds fo' the secono. _ _ 
:. Write /draU first, and tnen cop, it over leqioiv. 



Hppendi ' V 

^Pve: Agree'nenf ^"or Points Selected for fey 
Hebre. Te-..: "The F.l. h.d.a ^ro. . Flay Th^ng 

i. ' i.-^,<«rr. T-rsp'M Frcse trofTi Detachffient tc 
'■The Problefftaticitv o- hoaern i^rdGi] frt-e 

Continui tv *' r. ,r o - R/Q 

1. S/9 7/^ :■• 7/9 5. c. -'/9 8/9 

"H3« to HVDid Fool.sh Opinions" r ,d - o/c 

; :. 6/9 :. :/9 4. 9/= :.. '/■J o. B/9 8' 

Miodern Constitutions' ^ 
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